Skip to content

Prepare tests for downstream - #334

Merged
Jakob-Naucke merged 8 commits into
trusted-execution-clusters:mainfrom
Jakob-Naucke:test-for-downstream
Aug 14, 2026
Merged

Prepare tests for downstream#334
Jakob-Naucke merged 8 commits into
trusted-execution-clusters:mainfrom
Jakob-Naucke:test-for-downstream

Conversation

@Jakob-Naucke

@Jakob-Naucke Jakob-Naucke commented Aug 11, 2026

Copy link
Copy Markdown
Member
  • Allow for Makefile to be included from elsewhere
  • Allow for setting an upstream directory other than ..
  • Make get_encoded_root_pem, get_env public
  • Make TestContext clonable
  • Use TestContext.info when possible to save lines
  • a rename

Preview of downstream test to use these changes: https://github.com/Jakob-Naucke/confidential-clusters-operator/tree/scale-test

Summary by Sourcery

Prepare test utilities and build tooling to be reusable from downstream projects and support multiple virtualization backends.

New Features:

  • Allow configuring the upstream workspace directory via the UPSTREAM_DIR environment variable when generating operator manifests.
  • Expose helper functions for environment variable access and root CA PEM encoding for reuse across modules and downstream tests.
  • Introduce a shared NodeBackend trait to abstract SSH execution and root key retrieval across different VM backends.

Enhancements:

  • Make TestContext clonable and consolidate logging through its info method to reduce verbosity in tests and utilities.
  • Refactor VM backend configuration to pass an already encoded CA PEM into ignition generation instead of encoding it per backend.
  • Update kubevirt and Azure backends to implement the new NodeBackend trait and use the shared shell execution helper.
  • Adjust internal APIs (e.g., get_root_key signature and ssh_exec renamed to sh_exec) to better fit shared backend abstractions.

Build:

  • Make the Makefile self-contained by resolving paths relative to the Makefile directory, enabling inclusion from other repositories and fixing version resolution for Go and Cargo tools.
  • Update trusted-cluster-gen build target to work correctly when the Makefile is invoked from external directories.

Tests:

  • Adapt attestation tests to the updated VM backend interface, passing Kubernetes client and namespace to retrieve root encryption keys.

@sourcery-ai

sourcery-ai Bot commented Aug 11, 2026

Copy link
Copy Markdown

Reviewer's Guide

Prepares the test utilities and Makefile for use from downstream projects by making key helpers public, parameterizing paths, refactoring VM backends through a shared node interface, and consolidating logging and TLS handling.

File-Level Changes

Change Details Files
Expose and refactor common test utilities to support downstream consumers and shared TLS handling.
  • Made get_env public so downstream code can reuse environment-variable validation.
  • Added UPSTREAM_DIR_ENV and used it to parameterize the workspace root when generating manifests, falling back to the previous default.
  • Introduced get_encoded_root_pem to fetch and percent-encode the root CA secret, returning a data: URL string for reuse.
  • Imported APPROVED_IMAGE_NAME constant and used it in logging instead of fully-qualified references.
test_utils/src/lib.rs
Make TestContext clonable and streamline logging using its info helper.
  • Derived Clone for TestContext and ensured test_namespace is cloned where needed.
  • Replaced many direct test_info! macro invocations with TestContext.info calls, including namespace lifecycle, manifest generation, RBAC preparation, deployment readiness, and ConfigMap/ApprovedImage waits.
  • Minor formatting adjustments to use captured variables in log messages consistently.
test_utils/src/lib.rs
Refactor VM backend layer to introduce a NodeBackend trait, centralize SSH command execution, and move root key retrieval into the shared node abstraction.
  • Renamed ssh_exec helper to sh_exec to reflect its generic shell usage and kept its process-handling semantics.
  • Added NodeBackend trait with ssh_exec and get_root_key implementations that operate against a Kubernetes Secret via UUID extraction from VM, using anyhow::Context for better error messages.
  • Updated VmBackend trait to extend NodeBackend, keeping VM lifecycle and SSH readiness helpers while delegating SSH and root-key logic to the new trait.
  • Adjusted VmConfig to carry an encoded CA PEM string instead of raw PEM and updated generate_ignition to consume the pre-encoded data URL.
  • Updated create_backend to call get_encoded_root_pem, remove inline secret parsing, and populate encoded_ca_pem in VmConfig.
test_utils/src/virt/mod.rs
Adapt Kubevirt and Azure backends to implement NodeBackend and reuse shared SSH and root-key logic.
  • Implemented NodeBackend for KubevirtBackend, building the virtctl ssh command and delegating execution to sh_exec.
  • Removed KubevirtBackend-specific get_root_key implementation, relying on NodeBackend’s shared version instead.
  • Implemented NodeBackend for AzureBackend, resolving the public IP via az CLI then delegating SSH execution to sh_exec.
  • Provided a no-op get_root_key implementation for AzureBackend that returns None, matching its capabilities.
  • Updated both backends to use the new sh_exec helper and updated imports accordingly.
test_utils/src/virt/kubevirt.rs
test_utils/src/virt/azure.rs
Update attestation tests to use the new NodeBackend interface that requires Kubernetes client and namespace for root-key retrieval.
  • Changed SingleAttestationContext and parallel attestation tests to call backend.get_root_key(client.clone(), &namespace) instead of the previous parameterless version.
  • Ensured warnings about unencrypted root are preserved when root keys are unavailable for one or both backends.
tests/attestation.rs
Make the Makefile include-safe and robust when used from downstream directories.
  • Introduced MAKEFILE_DIR to represent the directory of the included Makefile and reused it for LOCALBIN so tools install in a stable, include-safe location.
  • Wrapped go list and cargo metadata invocations with cd $(MAKEFILE_DIR) so version detection works when the Makefile is included from another repository or directory.
  • Adjusted trusted-cluster-gen target to reference the source file relative to MAKEFILE_DIR and build from that directory, writing the resulting binary into the including project’s current directory.
Makefile

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="test_utils/src/virt/mod.rs" line_range="172-177" />
<code_context>
     }
 }

+#[async_trait::async_trait]
+impl NodeBackend for AzureBackend {
+    async fn ssh_exec(&self, command: &str) -> Result<String> {
</code_context>
<issue_to_address>
**suggestion (testing):** Add tests for `NodeBackend::get_root_key` UUID extraction and error paths

The updated `NodeBackend::get_root_key` centralizes UUID extraction and secret lookup, but key edge cases aren’t covered by tests: empty UUID output, SSH/`cryptsetup` failure, missing secrets, and secrets without a `root` key. Please add tests that mock `ssh_exec` and the Kubernetes client to cover: (1) a successful case where `root` is returned; (2) failure when the UUID output is empty; and (3) failure when the secret or `root` key is missing. A test-only backend is fine as long as these behaviors are clearly defined for downstream attestation tests.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment on lines 172 to +177
#[async_trait::async_trait]
#[auto_impl::auto_impl(Box)]
pub trait VmBackend: Send + Sync {
async fn create_vm(&self) -> Result<()>;
async fn wait_for_running(&self, timeout_secs: u64) -> Result<()>;
pub trait NodeBackend: Send + Sync {
async fn ssh_exec(&self, command: &str) -> Result<String>;
async fn get_root_key(&self) -> Result<Option<Vec<u8>>>;
async fn cleanup(&self) -> Result<()>;

async fn get_boot_id(&self) -> Result<String> {
let id = self.ssh_exec("cat /proc/sys/kernel/random/boot_id").await?;
Ok(id.trim().to_string())
}
async fn get_root_key(&self, client: Client, namespace: &str) -> Result<Option<Vec<u8>>> {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (testing): Add tests for NodeBackend::get_root_key UUID extraction and error paths

The updated NodeBackend::get_root_key centralizes UUID extraction and secret lookup, but key edge cases aren’t covered by tests: empty UUID output, SSH/cryptsetup failure, missing secrets, and secrets without a root key. Please add tests that mock ssh_exec and the Kubernetes client to cover: (1) a successful case where root is returned; (2) failure when the UUID output is empty; and (3) failure when the secret or root key is missing. A test-only backend is fine as long as these behaviors are clearly defined for downstream attestation tests.

Comment thread test_utils/src/lib.rs Outdated
Comment thread Makefile
@SpaceFace02

Copy link
Copy Markdown
Member

nit: One of your commits name is [test_utils/virt: s/s(sh_exec)/\1/]. is this intentional?

@Jakob-Naucke Jakob-Naucke left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: One of your commits name is [test_utils/virt: s/s(sh_exec)/\1/]. is this intentional?

Yes, it's the sed syntax for replacing ssh_exec with sh_exec

Comment thread Makefile
@SpaceFace02

SpaceFace02 commented Aug 13, 2026

Copy link
Copy Markdown
Member

Yes, it's the sed syntax for replacing ssh_exec with sh_exec

Right :)

Can you confirm that the special characters in the commit message wouldn't cause issues in the CI or downstream? (It probably won't)

NVM, looks like the only disallowed character in a commit message is a NULL byte.

@Jakob-Naucke

Copy link
Copy Markdown
Member Author

/test azure-integration-test

not relative to working directory, for inclusion from other Makefiles

Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
Assisted-by: AI
for use from other crates

Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
for use from other crates

Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
to a new `NodeBackend` trait, namely, `ssh_exec`, `get_root_key`, and
KubeVirt's implementation of `verify_encrypted_root` (which Azure
overrides), for sharing in node-level tests.

Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
It really doesn't do SSH at all

Signed-off-by: Jakob Naucke <jnaucke@redhat.com>
@Jakob-Naucke

Copy link
Copy Markdown
Member Author

ugh, lief. I also didn't know you'd have the test with your PR once you'd run it

/test azure-integration-test

@Jakob-Naucke

Copy link
Copy Markdown
Member Author

still lief, but it seems better now

/test azure-integration-test

@Jakob-Naucke

Copy link
Copy Markdown
Member Author

still

/test azure-integration-test

@Jakob-Naucke

Copy link
Copy Markdown
Member Author

@SpaceFace02 I'd live without passing Azure tests, does the comment LGTY?

@Jakob-Naucke

Copy link
Copy Markdown
Member Author

/test azure-integration-test

@openshift-ci

openshift-ci Bot commented Aug 13, 2026

Copy link
Copy Markdown

@Jakob-Naucke: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/azure-integration-test 566f553 link false /test azure-integration-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@Jakob-Naucke

Copy link
Copy Markdown
Member Author

Seemed like a delayed Trustee pod restart, so this should improve with #248

/test azure-integration-test

@openshift-ci openshift-ci Bot added the lgtm label Aug 14, 2026
@Jakob-Naucke
Jakob-Naucke merged commit 60125df into trusted-execution-clusters:main Aug 14, 2026
13 of 15 checks passed
@Jakob-Naucke
Jakob-Naucke deleted the test-for-downstream branch August 14, 2026 08:50
@openshift-ci

openshift-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: alicefr, Jakob-Naucke, SpaceFace02

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants